A motion compensated video interpolation is used for improving the quality of a video, e.g. for providing judder-free video as one example in the case of video processing or for reducing data volumes, for instance in the case of video compression. FIG. 1 illustrates a method of motion compensation video interpolation 100. Motion compensation video interpolation 100 generally consists of a motion estimation 101 and motion compensation 103. Video input 102 is stored in a video frame memory 105 storing frames 104, 106, 108, 110 of the video input 102. Motion estimation 101 and motion compensation 103 provide an interpolated video 112 as output by processing the frames 104, 106, 108, 110 of the video input 102.
A motion compensated video interpolation generally is followed by a motion estimation (ME) step. There are lot of ME block based algorithms, e.g. block-matching, three-dimensional recursive search (3DRS), Optical Flow, Phase Correlation, etc. These ME algorithms can be used with a motion compensated video interpolation. A motion estimator calculates a motion vector for each block of the pixels of a frame to interpolate, the image being temporally positioned between two neighboring frames of a digital television sequence. Block of pixels must contain at least one pixel. A motion vector is calculated for each of interpolated block of pixels of the image from the motion vectors that have already been calculated blocks of pixels and from the projection of a motion vector calculated for the previous source image and including the considered pixel. A motion compensated video interpolation generally requires true motion vector. Block based motion estimation leads to blocking artifact.
To reduce the block artifact an overlapped block motion compensated scheme (OBMC) is used. OBMC is briefly described in the following example. In the example, 9 blocks are involved in OBMC being indexed by variable 1<=i<=9. Each block is of size 8*8. The upper left block is marked by index i=1, the central block by i=5 and the lower right block by index i=9.
In block motion compensation (BMC) image reconstruction, each block is associated with one motion vector and pixel prediction is obtained by copying and pasting the 8*8 block from the previous frame offset by the motion vector. No block overlapping happens in BMC during the copy-and-paste procedure. OBMC uses the enlarged block, called the domain of window support that is of size 16*16. A 16*16 window support function is described by the bilinear shape.
The OBMC linear estimator for a prediction pixel intensity Ip can be written as
                                                        I              p                        ⁡                          (                              x                →                            )                                =                                    ∑                              i                =                1                            9                        ⁢                          [                                                                    W                    i                                    ⁡                                      (                                                                  x                        →                                            b                                        )                                                  ·                                                      I                    ^                                    ⁡                                      (                                                                  x                        →                                            -                                                                        u                          →                                                i                                                              )                                                              ]                                      ,                            (        1        )            where x is the absolute pixel position with respect to the frame coordinates, {right arrow over (x)}b is the pixel position relative to the coordinates of the current block within a block, {right arrow over (u)}i is motion vector of block i, Wi is the weighting coefficient of block i, Î is the previous reconstructed pixel intensity.
“H. Watanabe and S. Singhal “Windowed motion compensation”, Proc, SPIE's Visual Comm. Image Processing VCIP, vol. 1605, pp 582-589″ describes a method using the classical BMC motion search and applying OBMC only in reconstruction. This method has artifacts for motion vectors that cross interpolated pixels in object and/or uncovered areas.
For the copy-paste procedure in the majority of the cases a bilinear interpolation is used.
In the case of a frame to interpolate Iα between a previous frame Iold and a current frame Inew, the previously defined motion vector points to a frame point of each frame. This frame point may either be a pixel, or a point located between 4 pixels. The bilinear interpolation is performed with 8 pixels, i.e. 4 pixels of the previous frame and 4 pixels of the current frame according to equation (2)
                                          (                                          ∑                                  i                  =                  1                                8                            ⁢                                                α                  i                                *                                  V                  i                                                      )                    /                                    ∑                              i                =                1                            8                        ⁢                          α              i                                      ,                            (        2        )            
where V1, V2, V3 and V4 designate pixels of the previous frame and V5, V6, V7 and V8 designate pixels of the current frame and α1, . . . , α8 are weighting factors representative of the proximity of said pixels to the image point. Taking into account that motion estimation is never performed without errors, this type of compensated interpolation in movement contains visible defects in the image covered and uncovered areas. These defects disappear or appear between two successive frames since matching between pixels is not possible.
Frame interpolation algorithms are widely used at frame rate converters for creation of a judder-free video at a digital TV set, for example at 30 frames/second (sec) to 120 frames/sec. Real time frame motion compensation algorithms shall overcome the issues described in the following. A first issue is visible block artifacts because motion estimation has low spatial resolution, i.e. blocks. A second issue is the visible artifacts for covered and uncovered areas. The covered and uncovered areas disappear or appear between two successive frames since matching between these two frames is not possible. A third issue is optimization of algorithms for real-time hardware implementation.
U.S. Pat. No. 4,771,331 describes a motion compensating field interpolation method that allows the interpolation of several fields between every two transmitted fields of a digital television sequence. Using the model of translational displaced objects, a hierarchically structured displacement estimator is applied to cope with relatively large displacements. It provides a displacement vector with integer components for each picture element of the fields to be interpolated. A change detector is used to assure zero displacement vectors in unchanged areas. A two-coefficient spatial-temporal filter interpolates each picture element of the fields to be interpolated. This method has artifacts for motion vectors that cross interpolated pixels in object and/or uncovered areas.
U.S. Patent Publication No. 2007/0121725 describes a motion compensated frame interpolation apparatus and method that includes a motion vector interpolation module for an object and uncovered area to generate information for motion vectors that cross interpolated pixels in object and/or uncovered areas. A motion vector interpolation module for a covered area generates information for motion vectors that cross interpolated pixels in covered areas. A motion vector counter records a number of motion vectors that cross an interpolated pixel. A buffer for motion vector interpolation buffers the information for motion vectors that cross interpolated pixels in the object and/or uncovered areas and the covered areas. A video data interpolation module generates video data of the interpolated pixels in objects and/or uncovered areas and in the covered areas. No algorithm for detection of covered and uncovered areas is disclosed.
U.S. Pat. No. 8,223,839 B2 describes a motion compensated video interpolation that is not sensitive to the errors in the motion estimation vectors. The document proposes a motion compensated video interpolation method in which the interpolation step varies according to the amplitude and/or a reliability index of the estimated motion vector for the considered pixel. The number of pixels that is used for interpolation and the weighting factors associated with these pixels are made to vary according to the amplitude and/or reliability index of the estimated motion vector. No algorithm is disclosed for selection of neighbor pixels and thresholds for the amplitude of motion vectors.